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ABSTRACT 



This study tested the utility of an approach to analyzing 
second language learners' listening skills in the context of a classroom 
discussion ("two-way listening skills"). The approach used was the 
Communicative Outcome system, which shifts the focus in discourse analysis 
from counting of tokens of negotiation to evaluation of what participants 
actually achieved. The system was used to analyze routine classroom 
interactions of a native Japanese- speaking undergraduate student in a seminar 
skills course in English for academic purposes. The approach was found useful 
in underlining that learners’ performance should indeed be evaluated in 
qualitative terms (i.e., communication success) as well as quantitatively 
(i.e., in terms of presence or absence of meaning negotiation). The perceived 
value of the approach is used as a means for the teacher of gaining feedback 
on student learning, not for evaluation purposes. It offers a simple means of 
coding the process and outcome of learner-to-learner discussion, which 
teachers can use to monitor differences in the effectiveness with which 
learners deal with questions and discussion points, and then to provide them 
with feedback. (Contains 28 references.) (MSE) 
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QUESTIONS OF PRESENTATION: 
EVALUATING SUCCESS IN EAP SEMINAR SKILLS CLASSES 



Tony Lynch (IALS) 



Abstract 

In this paper I discuss my adaptation for teaching purposes of a research-oriented framework, 
the Communicative Outcome system ( Yule and Powers 1994), Having applied it to data from 
discussion episodes in English for Academic Purposes (EAP) seminar skills classes, I conclude 
that it offers the teacher a practical means of monitoring learners’ performances and of 
providing them with feedback. 

1. Introduction 



This paper stems from a pragmatic concern with helping teachers - primarily myself, at this stage - to 
recognise signs of learners’ improvement and to enable us to guide their efforts to become more 
effective listeners. There are plenty of sources of advice on ways of creating a supportive classroom 
atmosphere for teaching listening. Buck (1995) is perhaps the best article-length summary, and there 
are books such as Ur (1984), Anderson and Lynch (1988), Underwood (1989) and Rost (1990). But 
what is lacking in the methodological literature is a more detailed account of the tactical moves open 
to the teacher who wants to evaluate listening as it happens, in order to give feedback on 
performance. Those writers who have addressed issues of listening skill assessment, such as Rost 
(1990) and Thompson (1995), have tended to focus on the design of tests for summative evaluation - 
testing achievement at the end of a course - rather than for formative evaluation of the sort that 
Gillian Brown has argued for (e.g. Brown and Yule 1983, Brown 1986; 1995). Brown argues that 
teachers need a means of diagnosing what has caused learners to do poorly in specific listening tasks, 
in order to give them guidance and practice that will help them to do them better subsequently. 

Here I will be focusing on the assessment of two-way listening skills, rather than on one-way 
listening (e.g. to cassettes or broadcasts). This is for two reasons: firstly, it is easier to measure 
success on one-way tasks; secondly, I spend more time in the classroom trying to help students to be 
better interactive users of English than teaching one-way listening skills such as lecture note-taking. 
Let me expand a little on those two points. 

2. Assessing one-way listening 

The key reason for teachers’ greater confidence in our ability to assess one-way listening is the 
stability of the input. In a typical listening class, students listen to pre-recorded material on cassette, 
and the teacher works with an Authorised Version of the text, in the form of a transcript. If we know 
what was said and what was meant, then we can compare it with the students’ reconstruction of what 
was said - e.g. their notes, or answers to questions. However, assessing success in two-way listening 
activities is altogether more complex. First, the participants in EFL communication are, naturally, 
likely to be L2 learners, so that what they hear as listeners in an interactive task is less correct and 
less clear than the equivalent message produced by (most) native speakers. Secondly, in all but the 
most controlled tasks, the teacher cannot know in advance what the speakers are going to say and has 
to evaluate the listener’s response as it is produced in on-going discourse. 
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3. One two-way listening genre: the discussion 



The particular genre of two-way listening l want to consider is academic discussion. I am using the 
term 'discussion' in a highly specific sense, to refer to the segment of talk that follows the main 
speaker’s presentation in a seminar, when members of the audience ask questions and make 
comments about what the presenter has said. There are three aspects of this type of discussion that 
make it more difficult for the L2 learner than the classic paired information-gap task. Firstly, it 
involves a more complex pattern of turn-taking, with the initiative shifting from the presenter to the 
listener-who-asks-a-question or to the listener-who-follows-up-another’s-question. It is reasonable to 
assume that it is more difficult for a listener to bid for a turn among a group of fellow listeners than to 
be the sole partner in a pair task. 

Secondly, the content of the discussion is intellectually more demanding than the type of information 
often featured in paired tasks. The location of shops on a street map, or of garden furniture in a park, 
is of limited cognitive interest - though I fully recognise the value of such tasks in stimulating the 
type of talk that demands precision of expression and interpretation. As well as being more 
challenging, the content of academic discussions may take on a particular intrinsic value, since the 
students participating in an HAP seminar may want to grasp what a speaker is saying because the 
information is relevant to their own academic field. 

Thirdly, there is the socio-cultural dimension: individuals’ assumptions about appropriate seminar 
behaviour. Frequently students bring from their home culture negative attitudes to asking questions in 
a public setting; the very act of questioning is regarded in some cultures as a direct challenge to the 
speaker’s expertise (Pica 1987), or as an admission of ignorance or inattention (Rost and Ross 1991), 
rather than a positive indication of intellectual curiosity, as it tends to be in the Western academic 
context. (For further discussion of this issue, see Lynch 1994). 

Academic discussion, then, presents considerable difficulties for the listener. From the teacher's point 
of view, the main problem of judging the success of individual listeners’ interventions in discussion is 
bound up with the underlying purpose of the discourse. It represents what Duff (1986) called a 
divergent task, in which participants engage in interaction with no single predetermined solution. This 
is in contrast to convergent tasks, such as ‘Find the Difference’ or ‘Describe and Draw’, where those 
taking part have a shared goal and need to reach agreement. 

4. Evaluating success in academic discussion 

In the specific form of discussion I am concerned with, there is an additional factor that makes 
evaluation difficult: it is the learners, not the teacher, who decide both the form and the content of the 
input. This creates a communicative situation in which negotiation of meaning arises ‘naturally’, but 
in ways that are beyond the teacher’s control. So discussion is markedly different from the sort of 
information-gap tasks commonly featured in both LI and L2 communication research, which are 
designed to allow the researcher ‘privileged knowledge of the state of affairs which the speaker is 
trying to describe' (Brown, Anderson, Shillcock and Yule 1984: 76-77). 

I do not have space here to deal with negotiation of meaning in any detail; for recent reviews of 
research, see Lynch (1996) and Pica (1997). In brief, the main reason for interest in negotiation is the 
claim that the interaction modifications made in order to resolve comprehension problems may 
enhance eventual progress in the L2 (Long 1985). The precise psycholinguistic connection between 
short-term comprehension and long-term L2 learning remains a matter of dispute (e.g. van Lier 1990, 
Gregg 1993), although there is some evidence that negotiated leamer/teacher and leamer/leamer 
classroom communication can create the conditions for L2 progress (e.g. Pica, Lincoln-Porter, 
Paninos and Linnell 1996). 
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For my present purpose, suffice it to say that a great deal of time and effort has been devoted to 
devising tasks artfully constructed to require negotiation of meaning. However, it is precisely because 
so much emphasis has been placed on ‘channelling’ learners into negotiating meaning that there is a 
risk that when evaluating their performances, we will focus on the process of negotiation rather than 
the product achieved. To counter this, Yule and Powers (1994) proposed a means of assessing the 
quality of the product of information-gap tasks: the 'Communicative Outcome' system, shown below. 



Table 1 . Communicative Outcome: an assessment category system 



1 . No Problem : A problem exists but Is not Identified by either the sender or the receiver. 



2. Non-neqotlated Solutions 

a. Unacknowledged Problem : A problem is Identified by the receiver but not acknowledged by the sender. 

b. Abandon Responsibility : A problem Is identified by the receiver and acknowledged by the sender, but the sender 
does not take responsibility for solving the problem, either by saying they will skip it, leave it, never mind it or forget it, 
or by telling the receiver to choose any location or path. 

c. Arbitrary Solution : A problem Is Identified by the receiver and acknowledged by the sender who then makes an 
arbitrary decision about some defining feature of the location or path. The key element here is not accuracy, but the 
arbitrariness of the decision which does not attempt to take the receiver's world into account or to make the receiver's 
world match the sender's. 



3. Negotiated Solutions 

a. Receiver's World Solution : A problem Is identified and acknowledged by the sender who then tries to find out what is 
In the receiver's world and uses that Information to instruct the receiver, based on the receiver's perspective. 

b. Sender's World Solution : A problem Is Identified and acknowledged by the sender who then Instructs the receiver to 
make the receiver's world match the sender's, ignoring whatever information the receiver provides which does not fit 
the sender's perspective. 



Source: Yule and Powers (1994) 



The CO system, designed for research rather than for teaching, shifts the focus from the counting of 
tokens of negotiation to the evaluation of what the task participants actually achieved. From the point 
of view of the task designer, the most successful outcome is the negotiated ’receiver’s world’ solution, 
since that requires the speaker to take account of the listener’s perspective and reach a mutually 
satisfactory outcome. However, the system was intended to deal with interaction on the type of 
communicative task in which the designer controlled the distribution of information to maximise the 
potential for negotiation. As I have said, it is more difficult to assess performance on a divergent task 
such as discussion, and so I decided to see whether it is possible to simplify the CO framework to 
assess learners’ success in navigating their way through discussions, by reducing the evaluative 
decisions to two: Did the speaker and listener need to negotiate meaning? and Did they succeed in 
reaching a satisfactory answer? I realise that reducing the original system to two high-inference 
questions risks ending in oversimplification for the purposes of research, but I felt it was worth trying 
it out as a teaching aid. My adaptation takes the form of a four-cell matrix (Figure 1). 
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"1 2 
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Figure 1. The simplified Communicative Outcome matrix 

The cells are numbered to represent progress as a clockwise movement from 1 to 4. The least positive 
outcome is Cell 1 (non-negotiated communicative failure), with Cell 2 indicating an unsuccessful 
attempt at negotiation, and Cell 3 a successful negotiation; Cell 4 could represent the optimal 
outcome - mutual understanding without the need for negotiation. 

5. Evaluating a learner’s success 

I have tested the practicability of the matrix by using it to analyse the routine classroom recordings of 
‘Kazu’ (a pseudonym), a Japanese economics undergraduate who attended three weeks of the IALS 
pre-sessional EAP course, on which I taught one component of the course, an afternoon class in 
seminar skills. I had noticed from the start of the course that Kazu had much more difficulty than 
others in understanding spoken English, whether it came from a native speaker (e.g. the teacher’s 
instructions and recorded texts) or from his peers. This weakness was reflected in his score on entry 
(51% on a dictation test), which was the lowest in his class. I have discussed Kazu’s overall listening 
ability and possible reasons for his limited success elsewhere (Lynch 1997); what I would like to do 
here is examine his discussion performances in chronological order and categorise them in terms of 
the CO matrix. 

The seminar skills classes consisted of three sessions per week over three weeks - a total of 15 hours. 
The first four sessions comprised preparatory practice for seminars, based on a mixture of plenary, 
pair and group activities using published EAP materials (Lynch and Anderson 1992). Sessions 5-9 
were devoted to short seminars given by each student in turn, the first on either education or 
employment (core themes of the course), and the second on a specialist topic chosen by the student. 
Each seminar was in three stages: an initial presentation by the student; discussion of points and 
questions raised by the audience; and feedback given privately by the teacher. 

Stages 1 and 2 of all the seminars were tape-recorded for the purposes of feedback and review. Apart 
from the prepared presentations for his own seminars, which I will not discuss here, Kazu contributed 
to eight discussion episodes: two as a listener to other students’ seminars, and six as the speaker in 
response to questions about his presentation (see Table 2). 



6 

ERIC 



55 



Table 2. Distribution of Kazu’s contributions to discussion 



Session 


Content 


Kazu’s contributions 


1-4 


Preparatory practice 




5 


Core seminars 1-6: Education 


Episode 1 as listener 


6 


Core seminars 7-12: Employment 


Episodes 2-3 as speaker 


7 


Specialist seminars 1-4 


Episode 4 as listener 


8 


Specialist seminars 5-8 


None 


9 


Specialist seminars 9-12 


Episodes 5-8 as speaker 



Clearly, Kazu participated relatively little as an audience member. In fact, only one of his two 
‘listener’ episodes - the one in session 7 * involved initiating a question; his contribution in session 5 
was to join what I have called a ‘multilateral negotiation’ (Lynch 1995), that is, a discussion episode 
in which the listener’s question is misunderstood by the speaker and clarified by other participants. I 
propose now to look at four of the six episodes following his two presentations in sessions 6 and 9, 
and to categorise them in terms of the CO matrix. 



Episode 2 

Park: I didn't understand what you said about + 'the competitive rate* + for public officials 

Kazu: it's becoming higher 

Park: I don’t understand what + the ’competitive rate 1 is 

Kazu: uh competitive rate ? 

Park: hmhm + I don’t know what that is 

Kazu: the meaning of ? 

Park: yes 

Kazu: um + + it means there are + many peoples + many peoples want to become + + want to 

enter + want to take the entrance examination + for 

Nobu: public official ? 

Kazu: yeah 

Park: uhuh + so the competition is getting stronger ? 

Kazu: yes 
Park: right 

This followed Kazu’s presentation on graduate employment in Japan. Park, a Korean student, asked 
Kazu a question that appeared to be a request for clarification of content. In his next turn he made 
clear that what he wanted was an explanation of the term ‘competitive rate’ and, once that was 
established, Kazu provided Park with the answer he required, i.e. that more and more people are 
competing for civil service posts in Japan. I would place this in Cell 3 of the CO matrix 
(+ negotiation, + success). 
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Episode 3 



Paul: under this recession + after your graduation + what kind of strategy do you have? 

Ss: Oaugh) 

Kazu: uh ( laughs ) pardon me? 

Paul: what kind + after graduation + + 1 think you are student at university? 

Kazu: yes 

Paul: what kind + + 

Kazu: yes? 

Paul: and after your graduation + + what kind of strategies + what kind of plans + will you have 

under this recession? 

Kazu: (laughs) ah yes + I’m thinking about + oh + I’m also thinking about + taking an 

examination for city + officials 

Here Paul’s question was intended to be humorous; he wanted to know what Kazu himself proposed 
to do, in view of the bleak employment prospects he had outlined in his presentation. The joke was 
not lost on the other students, as their laughter shows. Kazu laughed, too, but apparently out of 
embarrassment, as he had to ask for help. Paul helpfully backtracked and reformulated ‘strategies’ as 
‘plans’. Kazu then laughed again, this time appreciating the joke and giving an appropriate answer. 
Another case of successful negotiation - Cell 3 on the matrix. 

Episode 4 

I'd like to make some comment 
please 

generally the climate of Scotland is said + severe + and awful 
yes that's + the usual case 

and I think one of the reason is that the wind is very strong 
yes it usually is 

because wind is very related with our temperature + our body temperature 
hm ? 

yes + yes + so 

uh ? + you mean body temperature ? 
yes 

human beings' temperature ? 



Kazu: 

Lian: 

Kazu: 

Lian: 

Kazu: 

Lian: 

Kazu: 

Lian: 

Kazu: 

Lian: 

Kazu: 

Lian: 
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Kazu: yes + so + 1 think + the main + uh + point + uh + + + main component of + + Scottish 
weather is wind 

Lian: well it can never be complete when you only talk about one element + in + in climate + so I 

cannot say which one is the main + + figure + or main element when we talk about climate + + 
and I do not believe that the wind + because we are talking about atmospheric elements + 
conditions + so the human bodies' temperature in fact has little influence on the overall 
atmosphere conditions like wind + I don't think so + + + ok thank you 

This was Kazu’s first attempt to venture a comment on a presentation - on climate, by a Chinese 
student, Lian. Given that it went badly awry - with Lian understanding the exact opposite of the 
cause-effect relationship that Kazu was suggesting - it is possible that it put Kazu off the idea of 
contributing to discussion in subsequent seminars; certainly he remained silent for the other three 
seminars in session 7, and throughout session 8. This episode I would categorise as Cell 2 
(+ negotiation, - success). 

Discussion episodes 5-8 arose after Kazu’s specialist-topic seminar on industrial policy in Japan. The 
first two were both successful after negotiation - multilateral in one case and bilateral in the other - 
and for reasons of space I will pass over them here. There then occurred the only example of an 
exchange that can be counted as Cell 4 (successful without the need for any negotiation). Jean asked a 
question about the transfer of employees between branches of the same company, which Kazu had 
mentioned are a feature of Japanese company strategy: 

Episode 7 

Jean: um + do they have meetings in which they discuss their problems of the transfers? 

Kazu: uh it depends on the case but it usually + um seldom happens such a meeting 

Jean: ok 

This exchange is perfectly straightforward and belongs in Cell 4 - successful communication without 
any need for negotiation. This makes it the sort of exchange which researchers into negotiation 
discard as uninteresting, but which teachers (and learners) would hope for more often! 

Episode 8 

Paul: I heard that the lifetime employment was now drooping in Japan + is that right ? 

Kazu: is ... ? 

Paul: dropping + + 

Lian: decreasing + + decreasing 

Nobu: decreasing 

Kazu: yes decreasing 
Paul: in Japan 

Kazu: yes? 

Paul: is it right ? 

Kazu: yes 
Paul: + + + thank you 



9 
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Again we have a case of multilateral negotiation, this time involving four students. But this one is 
more difficult to place on my CO matrix. It seems to be Cell 3, since Paul closed by thanking Kazu 
for his answer, but on the other hand Paul’s long pause before ‘thank you’ suggests that he had 
actually hoped for a fuller answer. I think he may have been inviting Kazu to expand on the topic, 
rather than simply to confirm that what he had heard about Japan was right. This may be a case of 
inter-cultural differences in pragmatic interpretation. 

6. Discussion 

I chose to focus on Kazu’s discussion performances to assess the feasibility of using the simplified 
CO matrix to evaluate success in the classroom. Inevitably, analysing the performance of one learner 
in this way leads us to speculate about the specific problems he encountered and their possible causes 
and remedies, but I would like to concentrate on the potential of the CO framework, rather than 
Kazu’s particular difficulties. Does the CO matrix reveal or confirm any progress over the three 
weeks? 

On measures of one-way listening used in the EAP course, Kazu did in fact make considerable 
progress (for detailed discussion, see Lynch 1997). However, if we plot his sequence of discussion 
episodes in seminar skills classes over the three weeks, there is no clear pattern of improvement in his 
two-way listening. Figure 2 shows the results of my analysis of the discussion episodes presented in 
the previous section. (Those in which he participated as listener are shown in italics, and those in 
which he was the presenter are shown in bold). 



negotiation 

+ 





4 8 

“1 


7 


J 

I 2 3 5 6 



Figure 2. Kazu’s success in discussion episodes 

If we consider Kazu’s overall ‘strike rate’ of successful communication, we can see that he did rather 
well: six of his eight episodes fall into the bottom row (+ success). Similarly, if we look for 
indications of willingness to negotiate, then he scores seven out of eight; only episode 7 required no 
negotiation. On the other hand, there is no clear evidence of any progress over the period of 
instruction. In the final session 9, for example, we can see that in the space of some 5 minutes he 
participated in four episodes (5-8) whose outcomes range across three different cells. So, in Kazu’s 
case, the answer to the question that led me to try out the matrix is that it provides no clear picture of 
progress - other than, perhaps, the fact that his only completely non-negotiated success (episode 7) 
occurred in the final afternoon session of the course. 

It could be that the matrix reflects an accurate picture and that Kazu actually made no clear 
improvement in two-way listening. If so, was he unusual? To see whether there was any evidence of 
general progress within his group, I plotted all the episodes for all the seminars - see Fig. 3 
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Figure 3. The class’s success in discussion episodes 

If we consider simply the amount of negotiation that went on, the two sets of seminars produced 
almost identical proportions in the right-hand cells (31% and 30%). This could be taken as a sign that 
little had changed over the week between the two sets of seminars. However, when we look at the 
ratio of success to non-success by comparing the upper and lower cells, there is some indication of a 
change - from roughly 3:1 (77% success against 23% non-success) in the core seminars, to 9:1 (91% 
against 9%) in the specialist seminars. On that basis we might justifiably claim that the students in 
Kazu’s group achieved greater success the second time round. 

That change may have occurred for reasons other than an improvement in listening, of course. It 
could have been due to production factors such as improved delivery of listeners’ questions or 
presenter’s responses; or to social factors, e.g. the group's greater familiarity and ease with each 
other. Alternatively, it could be related to the presenters’ expertise in their specialist topics; there is 
research evidence (Whyte 1995) that L2 learners’ oral performance can be enhanced when they are 
talking on a familiar topic. 

I find it interesting that the overall number of questions fell between week 2 and week 3, from 45 to 
33. We could interpret this positively, as a consequence of improvements in expression and 
presentation by speakers in the seminar talk. However, the discussion stages of the specialist 
seminars were in fact longer than those of the core seminars the week before, so that, although there 
were fewer questions, the presenters’ responses were longer and more complex. Again, this ties in 
with findings by Whyte (1995) and in earlier work by, for example, Woken & Swales (1989) and 
Zuengler (1993). On the other hand, the fact that listeners said relatively little in discussion in the 
specialist seminars could be a negative indication that the presenters' relative authority made listeners 
feel more inhibited about raising queries on subjects they knew less about. 

7. Conclusion 

My aim was to see whether the simplified negotiation/success matrix based on the Communicative 
Outcome system might offer teachers - rather than researchers - a workable instrument for assessing 
learners' performances. Trialling the matrix has underlined for me that we do indeed need to evaluate 
learners' performance in discussion in qualitative terms (communicative success) and not just from 
the quantitative p oint of view (the presence or absence of negotiation). Negotiation is a tool to use 
when necessary - a means to an end, rather than an end in itself. 





Two-way listening tasks based on learner- control led input have rarely featured in negotiation 
research, yet learning activities such as EAP seminars play an important part as rehearsal for 
(academic) life beyond the classroom. Teachers need some way of monitoring and commenting on 
learners' success in negotiating meaning in divergent tasks of this sort. It may be that the real value of 
the CO matrix will be as a means of feedback, rather than of evaluation. It offers a simple means of 
coding the process and outcome of leamer-to-leamer discussion, and teachers could use it to monitor 
differences in the effectiveness with which learners deal with questions and discussion points, and 
then to provide them with feedback. 

The way I envisage using the matrix is for the teacher to tape the learner’s performance and then give 
him/her both the cassette recording and also a completed matrix sheet, possibly with brief 
annotations. The learner would then listen to (or view) the recording and see whether their own 
perceptions of what was said and done meshed with the teacher’s version of the performance, as 
conveyed in the matrix. If the learner felt the teacher had misjudged part of the interaction, s/he 
would be able to check with the other student(s) involved in the exchange. Whether the matrix is 
completable under real-time constraints, and how much additional information the teacher might 
need to provide in order to guide learners' subsequent review of their recordings, remain to be 
investigated in classroom use, but I believe it is a potentially useful tool in helping learners cope with 
the complexities of two-way listening in a foreign language. 
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